On Static Verification of Puppet System Configurations

Puppet is a configuration management system used by hundreds of organizations to manage thousands of machines. It is designed to automate tasks such as application configuration, service orchestration, VM provisioning, and more. The heart of Puppet is a declarative domain-specific language that specifies a collection of resources (packages, user accounts, files, etc.), their desired state (e.g., installed or not installed), and the dependencies between them. Although Puppet performs some static checking, there are many opportunities for errors to occur. Puppet configurations are often underconstrained (to allow them to be composed with each other) and underspecified (to allow them to be applied to a variety of machine states), which makes errors difficult to detect and debug. Even if a configuration is bug-free, when a machine is updated to a new configuration, it is easy for the machine state and its configuration to “drift” apart. In this paper, we identify determinism as the essential property that allows us to reason about configuration correctness and configuration updates. First, we present a sound, complete, and scalable algorithm to verify that configurations are deterministic. Our approach is to encode configurations as logical formulas in an SMT solver and apply partial order reduction and program slicing to achieve scalability. We apply our tool to real-world Puppet configurations gleaned from open-source projects that suffered determinacy bugs. Second, we consider the configuration update problem: a live update from version 1 to version 2 does not have the same effect as applying version 2 to a new machine. By treating configurations as functions—which we can do after we verify that they are deterministic—we build a simple program synthesis tool to calculate an update to version 1 that has the same effect that version 2 would have had on the original machine.